Optimal deep neural networks for sparse recovery via Laplace techniques

This paper introduces Laplace techniques to design an optimal neural network for estimation of simplex-valued sparse vectors from compressed measurements. To this end, we recast the problem of MMSE estimation (w.r.t. a prescribed uniform input distribution) as centroid computation of some polytope that is an intersection of the simplex and an affine subspace determined by the measurements. Due to a specific structure, the centroid can be computed analytically by extending a recent result by Lasserre that facilitates the volume computation of polytopes via Laplace transformations. Interestingly, the computation of volume and centroid can be performed by a classical deep neural network comprising threshold functions, rectified linear (ReLU) and rectified polynomial (ReP) activation functions. The proposed construction of a deep neural network for sparse recovery is completely analytical, which allows for bypassing time-consuming training procedures. Furthermore, we show that the number of layers in our construction is equal to the number of measurements which might enable novel lowlatency sparse recovery algorithms for a larger class of signals than that assumed in this paper. To assess the applicability of the proposed uniform input distribution, we showcase the recovery performance on samples that are soft-classification vectors generated by two standard datasets. As both volume and centroid computation are known to be computationally hard, the network width grows exponentially in the worst-case. However, the width may be reduced by inducing sparse connectivity in the neural network via a well-suited basis of the affine subspace. Finally, we point out that our analytical construction may serve as a viable initialization to be further optimized and trained using particular input datasets at hand.